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Article Info ABSTRACT 

Article history: Pleasant or unpleasant feeling stimulated by taste usually expressed to describe 
the acceptance or rejection of food and beverages intake by a human. Since it 

Received Aug I, 2019 stimulates the emotional reactions, therefore it also induces other response 

Revised Oct 2, 2019 such as heart rate variations. Traditional sensory tools used only subjective 

Accepted Nov 6, 2019 measurement such as self-report, to estimate the feeling of tasting. This 


method sometimes failed to show some differences between the pleasant and 
unpleasant type of feelings unconsciously. Previous unconscious measurement 
Keywords: methods used the intrusive technique by placing some sensors in contact with 
the body, which may affect the results of sensory analysis. This study was 


Basi 
asic taste conducted to avoid the effects of using contact sensors and validate the 


Electrocardiogram contact-less method of monitoring heart rate due to emotional changes by 
Image plethysmogram extracting plethysmographic signal from the green component of the video 
Noncontact measurement images. The videos were recorded while the subject responded to pleasant, 
Physiological signals unpleasant and neutral stimuli. The findings indicated that the heart rate was 
significantly related to taste stimuli that also reflected the subjective feelings. 
The unpleasant-taste influenced heart rate to increase more compared to 
pleasant-taste and neutral-taste. This proposed approach can be used to 
remotely detect the feeling/emotion that not overtly express through facial 

expression, speech or gestures. 
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1, INTRODUCTION 

Food and beverages appearance, aroma and taste give so much influence on a human to decide 
whether they like or not to consume the items. The sense of taste is essential for human to select good contents 
for ingestion by distinguishing so-called “basic” tastes; sweet, umami, sour, salty and bitter [1]. Recent studies 
have demonstrated the connection between food stimuli and emotions [2]. Each taste can induce pleasure or/and 
displeasure when various physiological responses occur associated with the gustatory stimuli. The change in 
the physiological signals can be used to relate the type of emotions evoked by taste [3-5]. Traditionally, the 
sensory evaluation commonly based on conscious and verbal/self-reported responses from participants to 
estimate their feeling of tasting and assess food acceptability by using only questionnaires or hedonic scales. 
By contrast, physiological responses analysis would instead be more appropriate to evaluate the sensational 
aspects due to taste perception, and it turns out to be a faster unconscious rating method [3]. Recently, heart 
rate has been used as one of the most common parameters to assess physiological responses associated with 
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consumption and visual assessment of food and beverages [6-8]. Immediate and automatic responses with none 
or very little time for thinking 1s a better decision-making process for selecting the most appropriate item. 

Most of the recent studies used the unconscious responses of the human body by integrating heart rate 
measurement with noninvasive multimodal techniques, such as brainwaves, body temperature, respiration 
levels, skin conductance, and facial expressions to assess the emotions elicited by taste. However, these 
methods are intrusive, discomfort and challenging to apply since they required electrodes to be attached to the 
body, such as on the hands, chest, and scalp [9-11]. These might not present accurate results, due to the 
possibility of affecting sensory, behavior and physiological responses such as increasing of arousal caused by 
the stress/anxiety of having sensors attached to the bodies and being monitored [7-8]. Therefore, the use of 
contact method is less appropriate because it could produce bias results in sensory analysis of food and 
beverages since the subjects will be aware of the measurements that may influence their heart rate and other 
physiological responses [12]. It also can cause skin irritation for a long monitoring session and a risk of losing 
contact when the subjects are moving. Recent advancements of measuring physiological signals by contact- 
less or remote methods have progressively developed since it is noninvasive, nonintrusive, simple and low- 
cost for measuring multiple physiological parameters [13-17]. The monitoring of heart rate using video analysis 
1s an alternative method developed based on the photoplethysmogram (PPG) concept. Instead of using red and 
infrared lights, the PPG signals can also be measured by extracting the green component, which is absorbed by 
hemoglobin and oxyhemoglobin at a relatively shallow part of the skin [18]. The changes in blood volume will 
be evident at green wavelengths, and thus the blood flow can be measured in the skin surface especially in the 
cheeks and forehead, where the highest blood flow usually happened. This blood flow event can be recorded 
by using a video camera and further analyzed using signal processing technique to convert into the heart rate 
value [19]. 

Most of the standard heart rate estimation using video camera only measures the heart rate without 
emotional influences. They did not emphasize on the evaluation changes of the heart rate as an emotional 
measure or the effect of stimuli on autonomic arousal. The most well-studied emotional expression methods 
by noncontact technique only based on facial expressions, eye gaze, body gestures, and speech signals. Thus, 
this study was aimed to investigate and validate the noncontact method in measuring the heart rate changes due 
to emotional reactions caused by pleasant and unpleasant feeling by capturing the light reflected from the 
subject’s face known as the image photoplethysmogram (iPPG). This study is meant to be used as a tool to 
identify and understand the emotional state of a person, even the emotion is not overtly expressed. 


2. RESEARCH METHOD 
2.1. Experiment Materials 

The subjects consisted of ten individual healthy university students aged around 21-23 years old. 
However, the data recorded from the two subjects cannot be analyzed due to the technical problems. All 
subjects were free from medications and illness, non-smokers, no allergy for the drinks offered and not 
suffering from smell or taste deficiencies. They were requested to refrain from eating or drinking for at least 1 
hour prior to the experiment session. The stimuli proposed for evoking flavor sensations were chosen based on 
an investigation performed by previous researchers [3, 5]. They defined the chocolate cream/sucrose as a sweet 
flavor and rated it as the most pleasant. On the other hand, the lemon homogenate/citric acid was considered 
as a sour flavor and rated as strongly unpleasant. Meanwhile, the flavorless item or mineral water rated as 
neither pleasant nor unpleasant and used as a control stimulus. Based on the information, the taste samples 
used in this experiment were including the solutions of milk chocolate (sweet flavor), lemon juice (sour flavor) 
and mineral water (flavorless). Specifically, the sweet solution was used for evoking pleasant sensation 
whereas sour solution for evoking unpleasant sensation. On the other hand, natural mineral water was served 
as the control stimulus and also used for mouth cleaning. 

The experiments were conducted in an environment-controlled laboratory in such that the temperature 
and humidity were set at ambient values. There were three tasting sessions; sweet-taste (pleasant), sour-taste 
(unpleasant) and water-taste (neutral). The subjects individually tasted each solution, approximately 20 ml in 
a session lasted about 50 s - 60 s. The heart rate at rest was monitored for 30 s before the presentation of taste 
solutions and it was stored separately. At the start, the subjects took a sip of the solution and left it in their 
mouth for about 10 s for a complete impregnation over the entire oral area before they swallowed it. They had 
to restrain the facial reactions and maintain to look directly at the video camera. They also placed their chin 
and forehead on a headrest to minimize the head movement. In the last part of each experiment, the subjects 
were requested to do self-report on the degree of liking or disliking of stimulus using hedonic scales. They 
identified the taste of the solutions and scored their feeling: 7, Very pleasant; 6, Moderately pleasant; 5, Slightly 
present; 4, Neither pleasant nor unpleasant; 3, Slightly unpleasant; 2, Moderately unpleasant and 1, Very 
unpleasant. The experimental procedures were repeated identically for the other type of solutions. 
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2.2. Recording System and Data Acquisition 

This contact-less method used only a CMOS video camera with a white LED light source placed 
approximately 70 cm in front of the subjects. They were facing a video camera (TGX02c; Baumer Co., Ltd) 
during the recording sessions. All videos were recorded in color image at 120 frames per seconds (fps) with 
440x400 pixel resolution. The reference heart rate was monitored simultaneously throughout the experiment 
using the 3-lead ECG electrodes connected to a BIOPAC System (MP150) with AcqKnowledge (v.3.9.1) 
software. All the recorded video files and ECG signals were further analyzed using MATLAB (ver. R2017b). 
The electrocardiographic R-waves were detected offline and the intervals between heartbeats were converted 
into heart rate in beats per minute (bpm). 


2.3. Heart Rate Estimation Method 

Figure 1 demonstrates the main idea to estimate the heart rate from a video image. The 
plethysmographic signal was obtained by averaging the pixel values of the selected region of interest (ROI) in 
each video frame. After that, the signal was further analyzed using a signal processing technique to eliminate 
noises before estimating the heart rate. 
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Figure 1. The basic process of heart rate estimation using video image 


The color intensity-based method for obtaining heart rate from a face image 1s presented in Figure 2. 
A box ROI in Figure 2(a) was manually selected in the area that consists of many facial skin pixels after 
smoothing the whole image using the median filter. The image photoplethysmographic (1PPG) signal in Figure 
2(b) was obtained by averaging the brightness of the green component at each pixel values inside the ROI. 





(a) Frame (Time) rime(s} “() 
BANDPASS Passband 


FILTER 


ae er 
me aayyyl 1 taal 


determination ... s . c . = 

iad | 
= col | @ Heart rate component | 
5 _ 1 \ | 


MMIII) ;-—-- co 
i i \ } I I I Wy | | I | | oan 


rimate) a ‘ 
(d) 


Figure 2. Heart rate estimation method 


For the heart rate extraction, some filter processing was performed to remove the noise due to the 
variation of lighting, body movement, breathing, and camera adjustment. A linear trend and abrupt changes 
were also removed from the pulse wave signal that was obtained as the time series of the average value. The 
heart rate computation can be grouped into time-domain analysis and frequency-domain analysis [20]. The 
signal was transformed into the frequency domain using the Welch method. By using the power spectral density 
estimation, the dominant frequency with the largest amplitude as in Figure 2(c) is defined as the average heart 
rate. For the time domain processing, a Butterworth filter with the pass-band of 0.7-2.0 Hz was applied to 
remove noise components except for cardiac frequency components. It fits the heart rate variability between 
40-120 bpm for healthy people [21]. The signal’s maximum point interval that is known as a beat-to-beat 
interval (BBI) in Figure 2(d) can be used to estimate the instantaneous heart rate by multiplying the reciprocal 
of BBI by 60. 
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3. RESULTS AND ANALYSIS 
3.1. Image Photoplethysmogram (iPPG) and Electrocardiogram (ECG) 

The heart rate variability affected by taste stimuli is demonstrated in Figure 3. It is an example of a 
subject’s heart rate when responding to a pleasant taste. It is meant to compare the data taken via a video camera 
(aPPG) with the ECG module. Both techniques have almost the same results except during the facial 
movements, which is around 14 s-21 s, where facial heart rate fluctuated and different from ECG values. This 
condition was happened due to the movement of the mouth area, including facial reacting while swallowing 
the solution. The data during this period are not reliable and not accurate. Therefore, the data during stimulation 
from facial video image that contaminated with facial movements/reactions should be discarded. The results 
from iPPG had no significant differences (P>0.05) with ECG before and after stimulation, as displayed in 
Figure 3(b). This, therefore, proved that the iPPG is an acceptable technique to measure the heart rate reacted 
to taste stimuli provided no facial reactions are involved. 
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Figure 3. (a) Heart rate measurement in response to pleasant taste using both ECG (Blue) and video 
camera (Red), (b) Average heart rate before and after stimulation 


3.2. Reference Heart Rate 

The reference heart rate was measured with a 3-lead ECG module in order to compare and validate 
the results measured using the noncontact technique. The heart rate variations and the percentage of the heart 
rate increase/decrease in response to pleasant (sweet), unpleasant (sour) and neutral (water) stimuli are shown 
in Figure 4. The pleasant and unpleasant stimuli significantly increased heart rate responses immediately after 
the presentation of the taste solutions, whereas the neutral stimulus decreased the heart rate concerning the pre- 
stimulus value. 
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Figure 4. ECG measurement, (a) Heart rate variations, (b) Heart rate changes (%) 


The mean heart rate at rest measurement using ECG was 79.2 + 1.6 (mean + SD) for the 30 s before 
the presentation of the taste solutions. This value is considered as 100%. The increase/decrease of the mean 
heart rate for every 5 s after the presentation of the solutions are shown in Figure 4(b). Basically, the heart rate 
started to increase during the stimulation period (in the first 10 s) while consuming pleasant and unpleasant- 
taste until 10 s later they reached the peak heart rate. The maximum increases 1n heart rate induced by pleasant 
and unpleasant-taste were 6.6% and 13.5% respectively. However, mineral water (neutral-taste) lowered down 
the heart rate by 6.2% after 10 s of presentation of the solution. These heart rate changes were compared with 
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the mean level of the heart rate at rest. The sour solution was rated as unpleasant taste and it induced the 
acceleration of heart rate immediately after 5 s of presentation of the solution. Meanwhile, the sweet solution 
also increased the heart rate but slightly lower than the sour solution. By contrast, the neutral solution decreased 
the heart rate immediately before it returned to pre-stimulus level (after 30 s). Overall, the average increases 
after-stimulation for neutral, pleasant and unpleasant-taste were -3.9%, 4.8%, and 13% respectively. Even 
though the effect of mineral water was inverted from the effect of sour and sweet stimuli, but it showed a 
significant change (P <0.05, F>Fcrit) before and after consuming it. The ANOVA test for all taste stimuli also 
proved that the mean heart rate after-stimulation had significant differences with pre-stimulation heart rate 
(P<0.05, F>Fcrit). The results are consistent with the previous finding where the heart rate increases induced 
by the water (neutral) and sucrose (sweet) were significantly lower than those induced by citric acid (sour) [3]. 


3.3. Facial Heart Rate/Image Photoplethysmogram (iPPG) 

In the case of monitoring heart rate using a video camera, the measurement on the forehead ROI also 
shows the changes after experiencing each taste stimulus compared to pre-stimulus condition (Figure 5). The 
facial heart rate or image Photoplethysmograms (iPPG) showed that the pleasant and unpleasant stimuli 
significantly increased the heart rate responses after the presentation of the taste solutions whereas, neutral 
stimulus immediately decreased the heart rate after consuming it. The mean heart rate at rest measurement 
using video camera was 78.1 + 2.3 (mean + SD). This value is considered as 100%. The increases/decreases of 
the mean heart rate for every 5 s after the presentation of the solutions are shown in Figure 5(b). The heart rate 
changes were compared with the mean level of the heart rate at rest. The results show the similar trends of 
heart rate variations as measured using ECG where they also started to accelerate during the stimulation period 
(in the first 10 s) for pleasant and unpleasant-taste until they reached the maximum heart rate of 7% and 14.2%. 
Whereby, neutral-taste lowered down the heart rate by 4.2% after 10 s of presentation of the solution. Overall, 
the average increases after-stimulation for neutral, pleasant and unpleasant-taste were -2.3%, 5.9%, and 12.6% 
respectively. All differences were considered significant at a level of P < 0.05 before and after-stimulation. 

The accuracy estimation of the average heart rate measured in different ROI concerning the root 
means square error and the correlation coefficient is displayed in Figure 6. The selected ROIs including right 
cheek, left cheek and forehead. The heart rate measurement seems to be more accurate in the forehead area, 
especially for the case of measuring the effect of an unpleasant solution. This because of the forehead area was 
less affected while the subject was swallowing the solutions. The correlation coefficient in Figure 6(b) also 
displayed the relationship between heart rate measured using ECG and video camera in the forehead that are 
significantly related for all type of solutions. 
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Figure 5. 1PPG measurement, (a) Heart rate variations, (b) Heart rate changes (%) 
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Figure 6. Accuracy estimation in different ROI, (a) RMSE, (b) Correlation coefficient 
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3.4. Discussion 

The purpose of this study was to characterize each basic taste by heart rate responses, which 
objectively reflect the subject’s emotional reactivity. The variations in the heart rate were induced through the 
autonomic nervous system [22]. This study found that the unpleasant sour taste provoked a negative feeling 
hence, activated stronger heart rate response compared to pleasant sweet taste that provoked a positive feeling. 
This may due to pleasant sweet taste is becoming a familiar and usual taste hence, can be considered that the 
human has become “habituated” to sweet taste [23]. The ANOVA test for all three types of taste also gave the 
values of F>Fcrit and P<0.05, which determined the significant differences between all heart rates before and 
after stimulation. The percentage of change in the heart rate triggered by unpleasant-taste solution increased 
by 13% (ECG) and 12.6% (iPPG) right after the stimulation. Meanwhile, the percentage of change in the heart 
rate triggered by pleasant-taste solution increased by 4.8% (ECG) and 5.9% (iPPG). By contrast, the percentage 
of change in heart rate triggered by neutral-taste decreased by -3.9% (ECG) and -2.3% (iPPG) after the subjects 
consuming the solution. That means the sour solution significantly affect the feeling compared to the sweet 
and water solution used in this experiment. This accordance with the previous study reported that the rat 
neonatal heart rate responses increased after stimulation with lemon [24-25]. Not to mention, this study also 
showed that the forehead was accurately measured the heart rate compared to the left cheek and right cheek. 

Overall, the magnitude of heart rate responses could reflect the acceptance (lower the heart rate 
activation associated with sweet and mineral water) or rejection (stronger heart rate activation associated with 
sour) situation. Based on the results, the magnitude of the heart rate response could reflect the pleasant or 
unpleasant feeling that can be used to relate with acceptance or rejection of any food, drinks or medications 
consumption as well. When consuming bad tastes, humans try to evoke the rejection signal by demonstrating 
a higher heart rate. They tend to react negatively in certain emotions like sadness, anger, surprise, disgust and 
scared. Whereas, when consuming good tastes, they demonstrated positive emotions of happy or neutral that 
does not change the heart rate higher than during negative feelings. 


4. CONCLUSION 

In conclusion, this study has validated the practicability of implementation of a contact-less technique 
using only a video camera to assess the sensory analysis related to taste. This technique replaced the traditional 
physiological sensors with a camera, which will increase the usability of the detecting system and will 
overcome the complication of using contact sensors. The results showed that the significant correlation exists 
between taste stimuli and heart rate that reflected very subjective feelings. The unpleasant-taste induced the 
negative feeling thus exhibited a stronger heart rate response with higher amplitude compared to others. 
Nevertheless, the neutral taste was induced neutral feeling and displayed very weak amplitude. Besides, the 
response shows the increasing pattern as well for pleasure feeling due to the excitement, but lower than the 
displeasure feeling. This study produced a technique that has the advantage to sense the feeling in an 
unconscious situation. This technique can be useful in the detection of emotional/feeling state of person even 
if he/she does not overtly express his/her emotion through gestures, speech or facial expressions. In the future, 
this technique can be integrated with other modalities, such as facial expressions and gestures in order to 
enhance the emotion detection method. The nonrigid facial movements/expressions could affect the result of 
facial heart rate. Therefore, this work will be continued by improvising a method to overcome the nonrigid 
movements, especially in the ROI area. Not to mention, the distinction between initial and subsequent 
responses discovered by this technique using facial heart rate may offer a new valuable perspective for sensory 
and consumers’ eating behavior research. Due to the inexpensiveness and the fact that this method 1s relatively 
easy to implement even using only a smartphone camera, this method has the potential to become widespread. 
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